Consistent Weighted Sampling

We describe an efficient procedure for sampling representatives from a weighted set such that for any weightings S and T , the probability that the two choose the same sample is equal to the Jaccard similarity between them: Pr[sample(S) = sample(T )] = ∑ x min(S(x), T (x)) ∑ x max(S(x), T (x)) where sample(S) is a pair (x, y) with 0 < y ≤ S(x). The sampling process takes expected computation linear in the number of non-zero weights in S, independent of the weights themselves. Sampling computations of this form are commonly limited mainly by the required (pseudo) randomness, which must be carefully maintained and reproduced to provide the consistency properties. Whereas previous approaches require randomness dependent on the sizes of the weights, we use an expected number of bits per weight independent of the values of the weights themselves. Furthermore, we discuss and develop the implementation of our sampling schemes, reducing the requisite computation and randomness substantially in practice.